1. Field of the Invention
The present invention relates to an image processing method and an imaging apparatus using the image processing method. Particularly, the present invention relates to an image processing method enabling various types of image processing to be performed at high speed, such as correction of an image having distortion due to distortion aberration of an imaging lens, an image having unnatural perspective distortion of a subject due to high-angle or overhead photography, generation of an image with a viewpoint changed, mirror-image conversion of a captured image, and electronic zooming, and also relates to an imaging apparatus using the image processing method.
2. Description of the Related Art
Imaging lenses typically used for cameras and the like have various types of aberration, such as spherical aberration, astigmatism and comatic aberration. Among them, aberration causing geometrical distortion of a captured image is called distortion aberration. As typical measures for reducing the distortion aberration, a lens exhibiting minimized distortion aberration is designed.
However, lenses having the distortion aberration as-is have been used for special applications. For example, in a monitor imaging apparatus that is attached to the back of a vehicle and that captures images for rear-viewing, a fish-eye lens or the like to capture images of a very wide range at a super-wide angle is used so that a pedestrian who recognizes the existence of the vehicle but approaches without expecting reverse of the vehicle can be found without fail. Also, in a monitor imaging apparatus that is placed at an entrance and that captures images of visitors, a fish-eye lens or the like having a super-wide shooting range is used so that images of a suspicious person can be captured even if the suspicious person who doesn't want to be displayed in a monitor is at a position in a slanting direction with respect to the imaging apparatus.
In those lenses used for such applications, higher priority is put on imaging at a super-wide angle rather than on correction of the distortion aberration, and thus even significant distortion aberration of the fish-eye lens is left as is in many cases. However, since the fish-eye lens covers a view field of nearly 180 degrees, a subject near the edge of an image captured by the lens is distorted to a large extent, and it may be difficult to determine what a subject in the image is: a person or a pole.
On the other hand, referring to FIGS. 7A and 7B, it has been becoming a common practice to attach an imaging apparatus 31 to a back 32 of a vehicle 30, capture images during a reverse, and display the images in a display apparatus so that the vehicle can be easily parked. FIG. 7C is an enlarged cross-sectional view illustrating an attachment state of the imaging apparatus 31 and FIG. 7D is an enlarged view of a position where the imaging apparatus 31 is attached. The above-described fish-eye lens or an ordinary wide-angle lens is used in the imaging apparatus 31 in this example. In many cases, the imaging apparatus 31 is attached to the back 32 of the vehicle 30 such that the lens of the imaging apparatus 31 is oriented to the ground in a slanting direction, as illustrated in FIG. 7C.
Therefore, even if an ordinary wide-angle lens is used in the imaging apparatus 31, images obtained therefrom to be displayed are high-angle shots, that is, images with a perspective effect in which a nearer object is represented as larger and a farther object is represented as smaller, as illustrated in (A) in FIG. 8. Note that, in FIG. 8, reference numeral 40 denotes a position where a concrete wall rises, 41 denotes wheel stoppers, and 42 denotes lines indicating a parking position. However, when a driver tries to park the vehicle by reversing it while watching such an image with a perspective effect, he/she cannot sensuously recognize the distance between the vehicle and the wheel stoppers 41, so that he/she eventually reverses the vehicle by watching the back side with his/her eyes with the window or door opened, without watching a display screen. Accordingly, neither the imaging apparatus nor the display apparatus is utilized.
Furthermore, when an imaging apparatus serving as a videoconference apparatus is placed at a height, for example, images captured by the apparatus and displayed on a screen are unnatural: the heads of people are large while the other parts from the shoulders become gradually small toward the feet. This may cause a viewer of the screen to feel uncomfortable.
In such a case, an imaging apparatus that captures an image formed through a lens as image signals from an imaging device is capable of correcting data of an image having distortion or a perspective effect, unlike a silver-salt camera.
Specifically, as illustrated in (B) in FIG. 8, image data is corrected so that nearer and farther objects are displayed in the same size, for example that the lines 42 are parallel to each other and that the distance between the vehicle and the wheel stoppers 41 can be recognized. Accordingly, the driver can sensuously recognize the distance between the vehicle and the wheel stoppers 41 at a glance, which is very convenient. Even if the imaging apparatus of a videoconference apparatus is placed at a height, the viewer of the screen comfortably watches well-balanced corrected images of people.
Furthermore, in the imaging apparatus (rearview display apparatus) 31 attached to the back of the vehicle 32, if an image with a squint angle according to the attached position of the imaging apparatus as illustrated in (A) in FIG. 9 is corrected to an image of a view from a rear window of the vehicle as illustrated in (B) in FIG. 9 or to an image in which nearer and farther objects have the same size as illustrated in (B) in FIG. 8 and if the corrected image can be displayed, the utility value of the imaging apparatus 31 increases.
In a so-called digital camera, various types of image processing may be demanded, for example, electronic zooming to zoom in an arbitrary portion (e.g., center) of a captured image as illustrated in (A) in FIG. 10 in the manner illustrated in (B) in FIG. 10; and mirror-image conversion to convert a captured image as illustrated in (A) in FIG. 11 to a mirror image as illustrated in (B) in FIG. 11. That is, in the case where a user takes an image of himself/herself by holding a digital camera in hand with the lens directed to himself/herself (self shooting), if the user moves the camera in such a direction that himself/herself will be positioned at the center of a field, the camera may be actually moved in such a direction that the user departs from the center of the field because the moving direction of the camera is opposite to the moving direction of the user in the field. However, such a mistake can be avoided by using the mirror-image conversion, which realizes matching between the both directions. Also, when self shooting is performed by using zooming-in by electronic zooming, a small action is amplified, and thus, when the mirror-image conversion is not performed and when the above-described both moving directions are opposite to each other, the user may be out of the field and it may be difficult to recover an original state. Such a trouble can be avoided by using the mirror-image conversion, and the user's position for shooting can be recognized more appropriately.
In the above-described case where an imaging lens has distortion aberration or where an image captured by an imaging apparatus placed at a height has distortion due to a perspective effect, the image can be corrected in the following way. First, the correspondence between (i) the positions where respective points on a subject actually form an image by distortion aberration or high-angle photography and (ii) the positions of an image to be formed without distortion aberration or the positions of an image to be formed with a perspective effect being corrected is examined. Then, when the image data is output from an image data memory storing captured image data obtained from the imaging device, calculation is performed to determine the address in the image data memory of the captured image data to be read to output data in which aberration and a perspective effect in the respective points on the subject have been corrected in units of output addresses. Accordingly, output image data in which distortion and a perspective effect resulting from an imaging lens have been corrected can be obtained. This is the same for the cases of the above-described electronic zooming and mirror-image conversion illustrated in FIGS. 10 and 11.
For the purpose of the above-described operations, the following fish-eye lens camera apparatus using an image distortion correcting method and an image extracting method for the apparatus have been known. That is, an imaging apparatus provided with a fish-eye lens is attached at an arbitrary angle. The apparatus has a configuration to perform calculation by combining coordinate transformation to correct a perspective effect caused by the angle at which the imaging apparatus provided with the fish-eye lens is attached, and coordinate transformation to correct distortion of an image captured through the fish-eye lens. In this apparatus, high-speed map transformation can be performed on an image captured through the fish-eye lens of equisolid angle projection, a feature value of a figure or the like can be extracted by performing weighting corresponding to an area in the image captured through the fish-eye lens, and a display area can be extracted.
Typically, this process is performed by performing raster scanning on an output (write) image memory, obtaining the address of an input (read) image memory corresponding to the address of the output image data memory, and writing image data read from the input image memory of the address to the output image data memory. At this time, the address of the input image memory corresponding to the address of the output image data memory is not always an integer, and thus interpolation is required. However, since recent imaging devices have a very large number of pixels, such coordinate transformation in real time causes an enormous amount of calculation. Thus, a high-speed device is required, resulting in an expensive imaging apparatus.
Under these circumstances, the following image converting circuit to perform parallel movement, rotation, zoom in/out, and nonlinear geometrical (coordinate) transformation of an image at high speed has been known. That is, the image converting circuit includes an input image memory to accumulate original image data to be processed, an output image data memory to accumulate image data after transformation, an address calculating circuit to perform coordinate transformation on addresses in raster scanning of the output image data memory and calculate the addresses of the input image memory, and two look-up table circuits that receive two integers constituting each address output from the address calculating circuit, respectively, and that performs data conversion in accordance with a preset conversion characteristic. By using the data obtained from the two look-up table circuits as addresses, data is read from the input image memory, and the read data is written in the output image data memory.
Also, the following imaging apparatus for enabling automatic correction of distortion caused by inclination of the imaging apparatus has been known. The imaging apparatus includes inclination measuring means for measuring inclination of the imaging apparatus, electronic zooming means capable of changing magnification of a captured image, and a look-up table in which magnification is set with respect to the focal length, inclination of the imaging apparatus and the coordinates of an image. In this apparatus, magnification is determined by referring to the look-up table, the electronic zooming means is controlled by magnification control means, and a perspective effect resulting from the inclination measured by the inclination measuring means is corrected.
As described above, the amount of calculation can be reduced by preparing a look-up table for faster processing. However, when the number of pixels is very large, the size of the look-up table is also large and the cost increases accordingly.
An image processing method for performing various types of image processing at low cost and at high speed and an imaging apparatus using the method have been demanded.